end-to-end tests: refactor and fix flaky tests by qdm12 · Pull Request #2470 · ChainSafe/gossamer

qdm12 · 2022-04-06T13:28:50Z

💡 Review commit by commit by starting with the first commit

Changes

TLDR

Fix flakky tests and add testing depth
More robust code
Detect runtime errors
Better concerns separation
Can easily run a single test independently & locally (after make gossamer)

Details

Faster TestAuthorSubmitExtrinsic
Fix TestSync_SingleBlockProducer (Fix Flaky Tests in TestSync_SingleBlockProducer #2167)
refactor compareBlocksByNumber
refactor TestSystemRPC (Fix Flaky Tests in TestSystemRPC #2161)
- add empty skipped tests for missing cases
- split each subtest individually
- keep on retrying until main context is canceled
- fix networkState test case
- assert more fields
- fix Fix Flaky Tests in TestSystemRPC #2161 and RPC tests for system_health should start 2 peers and assert correctly #807
retry.UntilNoError function
- Use in compareBlocksByNumber
- Use in waitForNode
- change GetBlockHash to not retry RPC
- remove PostWithRetry
remove global variables maxTries and testTimeout in stress tests
refactor getResponse
- not test aware (tests/utils: make helping functions not test aware #2389)
- get passed target interface
- push test assertion/skip to actual test
rework start and stop of nodes (tests/utils: refactor nodes starting and stopping and detect runtime errors #2391)
- Remove unneeded logs
- Prefix writer with node string for multiple nodes
- Test writer using t.Logf
- Wait for nodes after starting all nodes
- Log when node is ready only
- Init nodes in parallel
split tests/utils package
- tests/utils/rpc package
- tests/utils/websocket package
- tests/utils/config package
remove DecodeRPC_NT
remove HOSTNAME variable
inline RPC method strings
remove global variable GenesisTwoAuthsSecondaryVRF0_9_10
remove global configuration variables
- write no auth config to temp dir per test
- write no grandpa config to temp dir per test
- write no babe config to temp dir per test
- write grandpa config to temp dir per test
- write default config to temp dir per test
remove global framework variable
remove global variables GenesisDefault and GenesisDev
Add GenerateGenesisAuths function
use t.TempDir() for base paths
waitForNode context based function (tests/utils: add waitForNode context-based function to wait for a node #2390)
make RPC helpers not test aware (tests/utils: make helping functions not test aware #2389)
make DecodeRPC not test aware (tests/utils: make helping functions not test aware #2389)

Tests

RPC suite passing
Stress tests passing

Issues

See issues mentioned in changes above.

Indirectly fixed: #2165

Primary Reviewer

codecov · 2022-04-06T21:53:35Z

Codecov Report

Merging #2470 (c1271b2) into development (6d22c23) will increase coverage by 0.00%.
The diff coverage is 28.07%.

@@             Coverage Diff              @@
##           development    #2470   +/-   ##
============================================
  Coverage        57.34%   57.34%           
============================================
  Files              215      215           
  Lines            28417    28407   -10     
============================================
- Hits             16296    16291    -5     
+ Misses           10442    10440    -2     
+ Partials          1679     1676    -3

Flag	Coverage Δ
unit-tests	`57.34% <28.07%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dot/utils.go	`50.00% <ø> (+16.66%)`	⬆️
lib/runtime/constants.go	`0.00% <ø> (ø)`
lib/runtime/test_helpers.go	`7.94% <0.00%> (-0.65%)`	⬇️
lib/utils/utils.go	`17.48% <0.00%> (-0.20%)`	⬇️
dot/peerset/peerset.go	`49.34% <42.85%> (-0.23%)`	⬇️
dot/state/epoch.go	`49.53% <50.00%> (+0.23%)`	⬆️
lib/runtime/life/test_helpers.go	`100.00% <100.00%> (ø)`
lib/runtime/wasmer/test_helpers.go	`100.00% <100.00%> (ø)`
dot/network/host.go	`64.31% <0.00%> (-1.07%)`	⬇️
lib/trie/database.go	`50.30% <0.00%> (-0.93%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2fa5d8a...c1271b2. Read the comment docs.

…etry.UntilNoError`

- Use `Decode` from rpc package instead - Add fields to RPC server response struct

…der received`

Co-authored-by: Timothy Wu <timwu20@gmail.com>

- Add `fetchWithTimeout` helper function - Change test cases slices to individual subtests - Add `t.SkipNow()` to parent tests where all subtests are skipped - Make `TestChainRPC` event driven and increase testing depth - Add a few `TODO`s to increase testing depth - Run each query call in parallel

- Refactor websocket code - Assert things and increase test depth - Graceful shutdown of websocket

- Use Go to implement `integration-test-all.sh` directly in single 'stable' test - The test is fast and only does RPC calls, it should be part of the RPC workflow - Remove stable workflow from Makefile, github workflows directory and document

I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**. I have been working full time on Gossamer since October 2021, mostly on the state trie and storage. I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository. I am requesting to join the Fellowship at rank 1. ## Main contributions ### Gossamer - Fix memory leaks - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009) - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286) - Fix sync benchmark [#2234](ChainSafe/gossamer#2234) - Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661)) - Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370)) - State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272)) - State trie fixes and improvements - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223) - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384) - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725) - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081) - Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485)) - Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991) Ongoing work: - State trie lazy loading and caching - State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530)) ### Polkadot specification ➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)

qdm12 changed the title ~~Qdm12/tests refactor tip~~ end-to-end tests: refactor and fix flaky tests Apr 6, 2022

qdm12 force-pushed the qdm12/tests-refactor-tip branch 5 times, most recently from 23c4806 to cce2193 Compare April 6, 2022 21:28

qdm12 force-pushed the qdm12/tests-refactor-tip branch 9 times, most recently from 3287691 to d02eb44 Compare April 11, 2022 19:43

qdm12 marked this pull request as ready for review April 11, 2022 20:04

qdm12 requested review from EclesioMeloJunior, edwardmack, jimjbrettj, kishansagathiya, noot and timwu20 as code owners April 11, 2022 20:04

qdm12 force-pushed the qdm12/tests-refactor-tip branch 4 times, most recently from dc36f16 to bd18ecd Compare April 27, 2022 12:25

qdm12 mentioned this pull request Apr 27, 2022

RPC endpoint system_peers stuck at 0 best block number for 1 of 2 peers #2498

Open

qdm12 force-pushed the qdm12/tests-refactor-tip branch from bd18ecd to 85c38a5 Compare April 27, 2022 14:12

qdm12 mentioned this pull request Apr 28, 2022

fix(dot): no error logged for init check #2502

Merged

qdm12 and others added 23 commits June 10, 2022 19:31

Do not assert best block number and hash for now

d6f16d8

chore(dot): remove exportConfig and TestExportConfig

204ffb7

chore(dot): remove ExportTomlConfig and its test

aa0ab13

chore(e2e): install npm dependencies from Go code

5b1a38c

chore(tests/rpc): remove 1s sleep

ca61882

chore(tests/stress): replace compareFinalizedHeadsWithRetry with `r…

5ea98fe

…etry.UntilNoError`

fix(tests/utils/node): reset log before starting command

501cbb9

chore(tests): getters without Get prefix

33b68f5

chore(tests/utils/retry): shared common code

9f731fa

chore(tests): remove websocket package

a1708ee

- Use `Decode` from rpc package instead - Add fields to RPC server response struct

chore(tests/utils/rpc): change error wrapping to `malformed block hea…

47b2f98

…der received`

chore(tests/utils/rpc): fix the bloody body

d85fc1e

Return runtime error channels instead of injecting them

20ae819

Simplify MakeNodes to use New for each node

a298950

Init in parallel for all code paths

f360f5a

Update tests/utils/runtime/errors.go

cd04c44

Co-authored-by: Timothy Wu <timwu20@gmail.com>

Move errors.go from runtime to node package

3d82658

Unexport all errorsFanIn types and methods

6bbb186

Rename rpc/models.go to rpc/types.go

7bece96

TestRestartNode: helper stopNodes function

dffb642

Refactor TestChainSubscriptionRPC

ee48c78

- Refactor websocket code - Assert things and increase test depth - Graceful shutdown of websocket

Merge stable workflow in rpc workflow

f0e0187

- Use Go to implement `integration-test-all.sh` directly in single 'stable' test - The test is fast and only does RPC calls, it should be part of the RPC workflow - Remove stable workflow from Makefile, github workflows directory and document

qdm12 force-pushed the qdm12/tests-refactor-tip branch from ff87feb to f0e0187 Compare June 10, 2022 19:32

qdm12 added 2 commits June 10, 2022 19:57

Log out configuration file content on runtime error

b35877f

Restore sleep between launches for stable tests

c1271b2

qdm12 merged commit 6220402 into development Jun 10, 2022

qdm12 deleted the qdm12/tests-refactor-tip branch June 10, 2022 21:10

qdm12 mentioned this pull request Sep 29, 2022

Member Request polkadot-fellows/seeding#28

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

end-to-end tests: refactor and fix flaky tests#2470

end-to-end tests: refactor and fix flaky tests#2470
qdm12 merged 47 commits intodevelopmentfrom
qdm12/tests-refactor-tip

qdm12 commented Apr 6, 2022 •

edited

Loading

Uh oh!

codecov bot commented Apr 6, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

qdm12 commented Apr 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

TLDR

Details

Tests

Issues

Primary Reviewer

Uh oh!

codecov bot commented Apr 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qdm12 commented Apr 6, 2022 •

edited

Loading

codecov bot commented Apr 6, 2022 •

edited

Loading